Wide Dynamic Range Depth Imaging

ABSTRACT

Wide dynamic range depth imaging in a structured light device is provided that improves depth maps for scenes with a wide range of albedo values under varying light conditions. A structured light pattern, e.g., a time-multiplexed structured light pattern, is projected into a scene at various projection times and a camera captures images of the scene for at least the same exposure times as the projection times. A depth image is computed for each of the projection/exposure times and the resulting depth images are combined to generate a composite depth image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims benefit of U.S. Provisional Patent ApplicationSer. No. 61/846,788, filed Jul. 16, 2013, which is incorporated byreference herein in its entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

Embodiments of the present invention generally relate to wide dynamicrange depth imaging in a structured light imaging device.

2. Description of the Related Art

In structured light imaging devices, a projector-camera pair is used toestimate the three-dimensional (3D) depth of a scene and shape ofobjects in the scene. The principle behind structured light imaging isto project patterns on objects/scenes of interest and capture imageswith the projected pattern. The depth is estimated based on variationsof the pattern in the captured image in comparison to the projectedpattern.

The amount of light reflected by the objects in the scene variesdepending on object properties such as color, albedo (reflectance) etc.The light incident on the objects is a combination of ambient light andlight from the projector. For a fixed camera exposure time and projectorprojection time, some objects may be under-exposed and other objects maybe over-exposed. Darker objects tend to be under-exposed and theprojected pattern is not detected in these regions. Conversely, brighterobjects may reflect the projected pattern and the ambient light and beover-exposed, which makes the pattern undetectable in such regions.

SUMMARY

Embodiments of the present invention relate to methods and apparatus forwide dynamic range depth imaging. In one aspect, a method of imageprocessing in a structured light imaging device having a camera and aprojector is provided that includes generating a first depth image basedon a first at least one image of a scene, wherein the first at least oneimage is captured by projecting a structured light pattern into thescene by the projector for a first projection time and concurrentlycapturing the first at least one image by the camera for at least afirst exposure time equal to the first projection time, generating asecond depth image based on a second at least one image of the scene,wherein the second at least one image is captured by projecting thestructured light pattern into the scene by the projector for a secondprojection time and concurrently capturing the second at least one imageby the camera for at least a second exposure time equal to the secondprojection time, wherein the second projection time is different fromthe first projection time, and generating a composite depth image bycombining the first depth image and the second depth image.

In one aspect, a structured light imaging device is provided thatincludes a projector configured to project a structured light patterninto a scene, a camera configured to capture images of the scene, and amemory configured to store software instructions that, when executed byat least one processor in the structured light imaging device, cause amethod of imaging processing to be performed. The method includesgenerating a first depth image based on a first at least one image of ascene, wherein the first at least one image is captured by projecting astructured light pattern into the scene by the projector for a firstprojection time and concurrently capturing the first at least one imageby the camera for at least a first exposure time equal to the firstprojection time, generating a second depth image based on a second atleast one image of the scene, wherein the second at least one image iscaptured by projecting the structured light pattern into the scene bythe projector for a second projection time and concurrently capturingthe second at least one image by the camera for at least a secondexposure time equal to the second projection, wherein the secondprojection time is different from the first projection time, andgenerating a composite depth image by combining the first depth imageand the second depth image.

BRIEF DESCRIPTION OF THE DRAWINGS

Particular embodiments in accordance with the invention will now bedescribed, by way of example only, and with reference to theaccompanying drawings:

FIG. 1 is a block diagram of an example digital structured light device;

FIG. 2 is a flow diagram of a method; and

FIGS. 3A-3B, 4A-4C, and 5A-5C are examples.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

Specific embodiments of the invention will now be described in detailwith reference to the accompanying figures. Like elements in the variousfigures are denoted by like reference numerals for consistency.

As previously mentioned, in structured light imaging, darker objects ina scene may be under-exposed and brighter objects may be over-exposed,making the projected pattern undetectable in corresponding regions ofcaptured images. Not being able to detect the pattern adversely affectsthe quality of depth maps computed for the scene.

Embodiments of the invention provide for wide dynamic range depthimaging in a structured light device that provides improved depth mapsfor scenes with a wide range of albedo values under varying lightconditions. A structured light pattern, e.g., a time-multiplexedstructured light pattern, is projected into the scene at variousprojection times and a camera captures images of the scene at the sameexposure times. Using different exposure times improves the detection ofa light pattern in darker and brighter regions of the scene as shorterexposure times tend not to over-expose brighter regions and longerexposure times allow more light to be captured in darker regions. Adepth image is computed for each of the exposure/projection times andthe resulting depth images are combined to generate a composite depthimage. Thus, the composite depth image may include depths for brighterand/or darker regions that may not have had a detectable pattern ifdepth was computed from a single exposure image.

FIG. 1 is a block diagram of an example digital structured light device100 configured to use an adaptive structured light pattern. Morespecifically, the digital structured light device 100 is configured toperform an embodiment of the method of FIG. 2 to generate wide dynamicrange (WDR) depth images.

The digital structured light device 100 includes a structured lightimaging sub-system 102, an image processing component 104, an imagingcontrol component 105, a memory component 114, a depth computationcomponent 110, and an application component 112. The components of thedigital structured light device 100 may be implemented in any suitablecombination of software, firmware, and hardware, such as, for example,one or more digital signal processors (DSPs), microprocessors, discretelogic, application specific integrated circuits (ASICs),field-programmable gate arrays (FPGAs), etc. Further, softwareinstructions may be stored in memory in the memory component 114 andexecuted by one or more processors (not specifically shown).

The structured light imaging sub-system 102 includes an imaging sensorcomponent 106, a projector component 108, and a controller component109. The imaging sensor component 106 is an imaging sensor systemarranged to capture image signals of a scene and the projector component108 is a projection system arranged to project one or more structuredlight patterns, e.g., a time-multiplexed structured light pattern, intothe scene. The imaging sensor component 106 includes a lens assembly, alens actuator, an aperture, and an imaging sensor. The projectorcomponent 108 includes a projection lens assembly, a lens actuator, anaperture, a light source, and projection circuitry. The structured lightimaging sub-system 102 also includes circuitry for controlling variousaspects of the operation of the sub-system, such as, for example,aperture opening amount, exposure or projection time, synchronization ofthe imaging sensor component 106 and the projector component 108, etc.The controller component 109 includes functionality to convey controlinformation from the imaging control component 105 to the imaging sensorcomponent 106 and the projector component 108, to convert analog imagesignals from the imaging sensor component 106 to digital image signals,and to provide the digital image signals to the image component 104.

In some embodiments, the imaging sensor component 106 and the projectioncomponent 108 may be arranged vertically such that one component is ontop of the other, i.e., the two components have a vertical separationbaseline. In some embodiments, the imaging sensor component 106 and theprojection component 108 may be arranged horizontally such that onecomponent is next to the other, i.e., the two components have ahorizontal separation baseline.

The image processing component 104 divides the incoming digitalsignal(s) into frames of pixels and processes each frame to enhance theimage data in the frame. The processing performed may include one ormore image enhancement techniques such as, for example, one or more ofblack clamping, fault pixel correction, color filter array (CFA)interpolation, gamma correction, white balancing, color spaceconversion, edge enhancement, denoising, contrast enhancement, detectionof the quality of the lens focus for auto focusing, and detection ofaverage scene brightness.

The depth computation component 110 then uses the enhanced image data toperform the processing steps of an embodiment of the method of FIG. 2 togenerate a composite depth image. The composite depth image is providedto the application component 112 for further application specificprocessing.

The memory component 114 may be on-chip memory, external memory, or acombination thereof. Any suitable memory design may be used. Forexample, the memory component 110 may include static random accessmemory (SRAM), dynamic random access memory (DRAM), synchronous DRAM(SDRAM), read-only memory (ROM), flash memory, a combination thereof, orthe like. Various components in the digital structured light device 100may store information in memory in the memory component 114 as imagesare processed.

Further, the memory component 114 may store any software instructionsthat are executed by one or more processors (not shown) to perform someor all of the described functionality of the various components. Some orall of the software instructions may be initially stored in acomputer-readable medium such as a compact disc (CD), a diskette, atape, a file, memory, or any other computer readable storage device andloaded and stored on the digital structured light device 100. In somecases, the software instructions may also be sold in a computer programproduct, which includes the computer-readable medium and packagingmaterials for the computer-readable medium. In some cases, the softwareinstructions may be distributed to the digital structured light device100 via removable computer readable media (e.g., floppy disk, opticaldisk, flash memory, USB key), via a transmission path from computerreadable media on another computer system (e.g., a server), etc.

The imaging control component 105 controls the overall functioning ofthe structured light imaging sub-system 102. For example, the imagingcontrol component 105 may adjust the focus of the imaging sensorcomponent 106 and/or the projector component 108 based on the focusquality and scene brightness, respectively, determined by the imageprocessing component 104. The imaging control component 105 may alsocontrol the synchronization of the imaging sensor component 106 with theprojector component 108 to capture images of the scene with theprojected pattern. Further, the imaging control component 105 controlsthe projection times the projection component 108 uses for projectingthe structured light pattern, e.g., a time-multiplexed pattern, into ascene and the exposure times the imaging sensor component 106 uses forcapturing images of the scene as needed for performing an embodiment ofthe method of FIG. 2.

More specifically, the imaging control component 105 causes theprojection component 108 and the imaging sensor component 106 to operateconcurrently at various pre-determined projection and exposure times. Inembodiments of the invention, the particular projection and exposuretimes and the number of projection and exposure times used arepre-determined and may depend on the particular application of thestructured light device 100 and/or capabilities of the particularprojection component and imaging sensor component in the device. Forexample, the expected scene content, the expected ambient lightconditions, the projection times available for the particular projectioncomponent, and the exposure times available for the particular imagingcomponent may be considered in determining the number of projection andexposure times to be used and the particular projection and exposuretimes.

The application component 112 receives the depth image and performs anyadditional processing needed for the particular application of thedigital structured light device 100. The application component 112 mayimplement an application or applications that rely on athree-dimensional (3D) representation of a scene. For example, theapplication component 112 may be a 3D reconstruction application thatgenerates point clouds (a collection of x, y, and z coordinatesrepresenting the locations of objects in 3D space) from depth maps. Inanother example, the application component 112 may be an applicationthat creates 3D models for computer rendering, physical printing of 3Dobjects, or fault detection.

FIG. 2 is a flow diagram of a method for wide dynamic range (WDR) depthimaging that may be performed by a structured light imaging device,e.g., the digital structured light device 100 of FIG. 1. The methodassumes that a pre-determined number of pre-determined projection andexposure times is to be used for generating a WDR depth image.Considerations for selecting particular projection and exposure timesand the number of projection and exposure times are previously discussedherein. Further, the method assumes that a pre-determinedtime-multiplexed structured light pattern is used. The time-multiplexedstructured light pattern may be any suitable pattern for the particularapplication of the structured light imaging device.

Initially, the pre-determined time-multiplexed structured light patternis projected 200 into the scene by the projector of the structured lightimaging device at a pre-determined projection time and a camera in thestructured light imaging device with a field of view substantiallyoverlapping that of the projector concurrently captures images of thescene at an exposure time equal to the projection time. That is, theprojector projects each pattern of the time-multiplexed pattern into thescene at the pre-determined projection time and the camera captures animage of the scene containing the projected pattern at an exposure timethat is the same as the projection time.

After images are captured of each pattern in the time-multiplexedpattern, a depth image corresponding to the pre-determined projectiontime is generated 202 based on the captured images and the patternimages of the time-multiplexed pattern. Generating a depth image frommultiple images of a time-multiplexed structured light pattern is wellknown and any algorithm appropriate for the particular time-multiplexedpattern may be used.

A check is then made 204 to determine if the current pre-determinedprojection time is the last of the pre-determined projection times. Ifthe current pre-determined projection time is not the last one, then thedepth image generation steps (200, 202) are repeated to generate a depthimage at the next pre-determined projection time. If the currentpre-determined projection time is the last one, the depth images arethen combined 203 to generate a composite depth image (WDR depth image)that is output for further processing as per the particular applicationof the structured light imaging device. In some embodiments, the depthimages are combined to generate the composite depth image using a simple“Xor-like” operation. For example, if two images are to be combined, fora given location in the composite image, if a depth value is present inone image, use that value or else use the depth value from the otherimage, if present. If neither image has a depth value for the location,then no depth value is provided for the location. If there is a depthvalue for the location in both images, use either one because the depthvalues are identical. One of ordinary skill in the art will understandhow more than two depth images may be combined to generate the compositeimage.

FIGS. 3A-5C are examples illustrating the efficacy of an embodiment ofthe method of FIG. 2. For these examples, a time-multiplexed structuredlight pattern composed of multiple gray code patterns was used. FIGS. 3Aand 3B show images of a scene with both bright and dark objects capturedat two different projection/exposure times, 6 milliseconds (ms) and 33ms, respectively. These images illustrate the need for differentprojection/exposure times. With the shorter projection/exposure time(FIG. 3A), insufficient light is captured by the camera in the darkerregions such that the projected pattern is not detectable, and thusdepth cannot be estimated in those regions. Similarly, in the image withlonger projection/exposure time (FIG. 3B), the brighter region isover-exposed such that the projected pattern is not detectable, and thusdepth cannot be estimated in this region.

FIGS. 4A and 4B show depth maps (images) generated from each of thecaptured images of FIGS. 3A and 3B, respectively. As these figuresdemonstrate, with the lower projection/exposure time (FIG. 4A), thebrighter regions are reconstructed but the darker regions are lost dueto under-exposure. Similarly, with the higher projection/exposure time(FIG. 4B), the darker regions are reconstructed, but the brighterregions are lost due to over-exposure. FIG. 4C shows the composite depthmap generated by combining the depth maps of FIGS. 4A and 4B. Note thatthis composite depth map includes depths for the brighter and darkerregions. FIGS. 5A and 5B show point clouds generated from the depth mapsof FIGS. 4A and 4B, respectively, and FIG. 5C shows the point cloudgenerated from the composite depth map of FIG. 4C.

Other Embodiments

While the invention has been described with respect to a limited numberof embodiments, those skilled in the art, having benefit of thisdisclosure, will appreciate that other embodiments can be devised whichdo not depart from the scope of the invention as disclosed herein.

For example, embodiments have been described herein in which theprojected structured light pattern is assumed to be a time-multiplexedpattern. One of ordinary skill in the art will understand embodiments inwhich a different type of structured light pattern is used, e.g., afixed single pattern or a continuous pattern. For example, for singlepattern images, depth/disparity is typically determined by some form ofpattern matching, and thus the depths for different exposure values maybe different. In such cases, post-processing such as connectedcomponents is commonly used to eliminate incorrect estimates. Once theseinconsistent values are eliminated, linear combination can be used tocombine the depth images to generate a composite depth image.

In another example, embodiments have been described herein in which theprojector and the camera operate at the same projection and exposuretimes. One of ordinary skill in the art will understand embodiments inwhich the camera does not support some or all of the desired patternprojection times. In such embodiments, if the camera does not support aparticular projector projection time, the camera can be operated at anexposure time longer than the projector projection time. For example, ifthe camera supports a single 30 ms exposure time and a shorter patternprojection time, e.g., 10 ms, is desired for computing a depth image,the camera can operate at the 30 ms exposure time while the projectorprojects for 10 ms and then projects nothing for the remaining 20 ms.

Embodiments of the method described herein may be implemented inhardware, software, firmware, or any combination thereof. If completelyor partially implemented in software, the software may be executed inone or more processors, such as a microprocessor, application specificintegrated circuit (ASIC), field programmable gate array (FPGA), ordigital signal processor (DSP). The software instructions may beinitially stored in a computer-readable medium and loaded and executedin the processor. In some cases, the software instructions may also besold in a computer program product, which includes the computer-readablemedium and packaging materials for the computer-readable medium. In somecases, the software instructions may be distributed via removablecomputer readable media, via a transmission path from computer readablemedia on another digital system, etc. Examples of computer-readablemedia include non-writable storage media such as read-only memorydevices, writable storage media such as disks, flash memory, memory, ora combination thereof.

It is therefore contemplated that the appended claims will cover anysuch modifications of the embodiments as fall within the true scope ofthe invention.

What is claimed is:
 1. A method of image processing in a structuredlight imaging device comprising a camera and a projector, the methodcomprising: generating a first depth image based on a first at least oneimage of a scene, wherein the first at least one image is captured byprojecting a structured light pattern into the scene by the projectorfor a first projection time and concurrently capturing the first atleast one image by the camera for at least a first exposure time equalto the first projection time; generating a second depth image based on asecond at least one image of the scene, wherein the second at least oneimage is captured by projecting the structured light pattern into thescene by the projector for a second projection time and concurrentlycapturing the second at least one image by the camera for at least asecond exposure time equal to the second projection time, wherein thesecond projection time is different from the first projection time; andgenerating a composite depth image by combining the first depth imageand the second depth image.
 2. The method of claim 1, furthercomprising: generating a third depth image based on a third at least oneimage of the scene, wherein the third at least one image is captured byprojecting the structured light pattern into the scene by the projectorfor a third projection time and concurrently capturing the third atleast one image by the camera for at least a third exposure time equalto the third projection time, wherein the third projection time isdifferent from the first projection time and the second projection time;and wherein generating a composite depth image comprises generating thecomposite depth image by combining the first depth image, the seconddepth image, and the third depth image.
 3. The method of claim 1,wherein the structured light pattern is a time-multiplexed structuredlight pattern and wherein the first at least one image and the second atleast one image comprise an image for each pattern in thetime-multiplexed structured light pattern.
 4. A structured light imagingdevice comprising: a projector configured to project a structured lightpattern into a scene; a camera configured to capture images of thescene; and a memory configured to store software instructions that, whenexecuted by at least one processor comprised in the structured lightimaging device, cause a method of imaging processing to be performed,the method comprising: generating a first depth image based on a firstat least one image of the scene, wherein the first at least one image iscaptured by projecting a structured light pattern into the scene by theprojector for a first projection time and concurrently capturing thefirst at least one image by the camera for at least a first exposuretime equal to the first projection time; generating a second depth imagebased on a second at least one image of the scene, wherein the second atleast one image is captured by projecting the structured light patterninto the scene by the projector for a second projection time andconcurrently capturing the second at least one image by the camera forat least a second exposure time equal to the second projection time,wherein the second projection time is different from the firstprojection time; and generating a composite depth image by combining thefirst depth image and the second depth image.
 5. The structured lightimaging device of claim 4, wherein the method further comprises:generating a third depth image based on a third at least one image ofthe scene, wherein the third at least one image is captured byprojecting the structured light pattern into the scene by the projectorfor a third projection time and concurrently capturing the third atleast one image by the camera for at least a third exposure time equalto the third projection time, wherein the third projection time isdifferent from the first projection time and the second projection time;and wherein generating a composite depth image comprises generating thecomposite depth image by combining the first depth image, the seconddepth image, and the third depth image.
 6. The structured light imagingdevice of claim 4, wherein the structured light pattern is atime-multiplexed structured light pattern, and wherein the first atleast one image and the second at least one image comprise an image foreach pattern in the time-multiplexed structured light pattern.